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1.  Introduction 


1.1  Purpose 

The  purpose  of  this  paper  is  to  identify  command  and  control  (C2)  experimentation  lessons 
learned  from  the  data  collection  and  analysis  processes  of  the  US  Army  Division  XXI  Advanced 
Warfighting  Experiment  (DAWE).  These  lessons  learned  will  assist  the  analytic  and  data 
collection  communities  in  conducting  future  Advanced  Warfighting  Experiments  (AWEs)  and 
the  Army  Experimentation  Campaign  Plan  (AECP).  This  document  provides  a  background  of 
the  DAWE,  an  overview  of  the  data  collection  and  analysis  processes,  and  the  C2 
experimentation  lessons  learned  during  the  planning,  execution  and  post-event  phases  of  the 
DAWE.  The  lessons  learned  section  includes  examples  of  the  problems  encountered  and 
potential  solutions.  Lastly,  this  document  highlights  the  general  implications  resulting  from 
these  lessons  learned.  This  paper  is  an  edited  version  of  a  Technical  Document  (TRAC-TD  - 
0198)  published  by  the  TRADOC  Analysis  Center  in  November  1998. 

1 .2  Background 

The  DAWE  culminated  a  series  of  Advanced  Warfighting  Experiments  (AWEs)  conducted  over 
the  past  several  years  by  the  Army  to  underpin  the  definition  and  development  of  the  Force  XXI 
Army.  The  DAWE,  conducted  from  5-13  November  1997  at  Fort  Hood,  Texas,  employed  a 
Battle  Command  Training  Program  (BCTP)-like,  command  post  (CP),  warfighter  exercise 
(WFX)  with  digitized  division  and  brigade  tactical  operations  centers  (TOCs)  in  the  field.  The 
purpose  of  the  DAWE  was  to  experiment  with  a  Force  XXI-like  division  represented  by  the 
digitized,  modernized,  and  trained  experimental  force  (EXFOR)  within  a  realistic  warfighting 
environment.  The  4th  Infantry  Division  (mechanized)  (4ID)  was  designated  and  configured  as 
the  EXFOR  using  a  Force  XXI  division  design,  which  included  a  new  centralized  combat  service 
support  (CSS)  concept.  The  DAWE  was  portrayed  in  the  Lantica  scenario  (force  year  2003). 
The  National  Simulation  Center  (NSC)  ran  the  simulation  in  a  non-stop,  free  play  mode  to  drive 
the  AWE  using  the  Joint  Training  Confederation  (JTC)  of  models,  modified  to  the  degree 
possible  to  represent  the  modernized  division  and  its  operational  concept  and  to  stimulate  the 
Army  Battle  Command  System  (ABCS)  systems  used  by  the  EXFOR.  There  were  no  maneuver 
units  in  the  field.  The  focus  of  the  DAWE  was  on  battle  command  (BC)  and  information 
operations  (10).  The  principal  products  of  the  DAWE  analysis  are  the  Initial  Insights  Report 
(HR)  published  in  January  1998  and  the  DAWE  Final  Report  published  in  August  1998. 

The  DAWE  was  one  of  several  key  Army  efforts  within  the  Joint  Venture  (JV)  axis  of  Force 
XXI.  The  other  major  analysis  efforts  on  the  JV  axis  were  Division  Design  Analysis  (DDA), 
which  examined  alternatives  for  the  interim  division  design;  Task  Force  XXI  (TFXXI)  AWE, 
which  addressed  digitization  and  other  issues  at  brigade  level;  and  a  series  of  CSS  evaluations  to 
address  the  new  Force  XXI  CSS  concepts  and  enablers.  Each  JV  effort  had  a  different,  yet 
complementary  focus  and  approach  to  address  specific  areas  of  interest  regarding  the  future 
division.  Consequently,  the  DAWE  was  not  intended  to  address  every  issue  or  area  of  interest 
regarding  the  design  of  the  division. 


2 


2.  Data  Collection  and  Analysis  Processes 

This  section  provides  an  overview  of  the  data  collection  and  analysis  processes,  which  formed 
the  basis  for  the  lessons  learned  presented  later  in  this  report.  This  overview  encompasses  the 
DAWE  planning,  execution,  and  post-event  phases. 

2.1  Planning  Phase 

This  phase  encompasses  the  time  from  initiation  of  the  DAWE  process  until  the  execution  of  the 
exercise  in  November  1997,  a  period  of  over  two  years. 

2.1.1  Refining  the  Hypothesis 

The  DAWE  hypothesis,  with  its  focus  on  BC  and  10  contributions,  was  developed  by  the 
Training  and  Doctrine  Command  (TRADOC)  Joint  Venture  Office  (JVO),  in  conjunction  with 
the  EXFOR  Working  Group  (EWG)  chaired  by  the  TRADOC  Commander.  To  assist  in 
addressing  the  DAWE  hypothesis,  eight  subordinate  questions  were  developed  to  provide  a  focus 
and  framework  for  the  full  set  of  issues  and  initiatives  that  were  considered  under  the  DAWE. 
These  eight  questions  became  known  as  the  “Brown  Card”  questions  that  originated  from  the 
Joint  Venture  Analysis  Management  Plan  (JVAMP)  overarching  issues.  These  Brown  Card 
questions  became  the  issue  categories,  which  served  to  frame  the  data  collection  and  analysis 
processes. 

2.1.2  Identifying  Study  Issues  and  Initiatives 

Within  the  scope  of  the  hypothesis  and  the  Brown  Card  questions,  the  DAWE  provided  an 
opportunity  to  address  numerous  specific  study  issues  and  initiatives.  A  “study  issue”  is  a 
question  to  be  answered  through  application  of  analysis  and  an  “initiative”  is  a  system  and/or 
concept  to  be  evaluated  through  experimentation  for  potential  fielding.  Proponents  from  the 
TRADOC  battle  laboratories,  schools,  and  centers  (as  well  as  other  outside  agencies)  nominated 
study  issues  and  initiatives  that  were  then  screened  and  revised  by  a  review  board  in  August 
1996.  These  proponents  are  called  “issue  proponents  (IPs)”  in  the  remainder  of  the  paper.  The 
resulting  issues  and  initiatives  were  included  in  the  DAWE  Study  Plan.  A  second  review  board 
in  April  1997  reconsidered  the  DAWE  issues  and  initiatives  based  on  the  TFXXI  AWE  results. 
The  issues  and  initiatives  that  emerged  from  these  two  review  boards  were  then  addressed  in  the 
DAWE. 

2.1.3  Developing  the  Analysis  Approach 

The  analysis  of  the  DAWE  was  organized  and  conducted  from  two  principal  perspectives.  One 
was  from  the  perspective  of  the  division’s  overall  battle  effectiveness,  known  as  the  “battle 
analysis.”  This  was  also  referred  to  as  the  “horizontal  analysis”  because  it  spanned  the  entire 
division’s  operation  during  the  successive  battles  waged  during  the  exercise.  The  battle  analysis 
was  led  and  conducted  by  a  select  team  of  TRADOC  Analysis  Center  (TRAC)  military  analysts, 
known  as  the  Capabilities  and  Potential  (CAP)  team.  Through  partnering  with  BCTP  and  direct 
observation  of  the  DAWE,  the  CAP  team  identified  key  battles  and  events  of  the  DAWE 
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scenario  that  offered  the  most  fruitful  scenario  opportunities  for  analysis  of  the  capabilities  and 
potential  of  the  digitized  division.  The  CAP  team  then  examined  the  data  collected  and  insights 
generated  about  those  events  by  the  BCTP  observer  controllers  (OCs),  subject-matter  experts 
(SMEs),  and  the  individual  members  of  the  CAP  team.  An  SME  is  a  representative  from  a 
TRADOC  school  and  center  (or  other  outside  agency)  used  as  an  observer  under  the  control  of 
the  lead  agency  for  data  collection,  the  Army  Test  and  Evaluation  Command  (ATEC) 
Operational  Test  Command  (OTC).  ATEC  was  formerly  known  as  the  Operational  Test  and 
Evaluation  Command  (OPTEC).  With  this  information  and  the  assistance  of  the  BCTP  Battle 
Summary  Cell,  the  CAP  team  generated  a  holistic  understanding  of  the  events  within  the 
operation  and  translated  that  understanding  into  potential  Force  XXI  capabilities. 

The  second  perspective  was  from  that  of  the  eight  issue  categories  (Brown  Card  questions), 
referred  to  as  the  “proponent  issue  analysis.”  This  was  also  referred  to  as  the  “vertical  analysis” 
because  it  delved  more  deeply  into  each  of  the  eight  issue  categories  within  the  division’s 
operation.  The  proponent  issue  analysis  was  led  by  TRAC  and  conducted  by  IPs  and  analysts 
assigned  to  address  the  issues  and  initiatives  within  each  of  the  eight  categories. 

2.1.4  Developing  the  Data  Collection  Approach 

ATEC  (OTC)  consolidated  all  data  collection  requirements  into  a  data  collection  plan  which 
identified,  for  each  issue,  the  types  of  data  to  be  collected  and  the  corresponding  sources  of  data 
in  the  DAWE.  Principal  sources  included  SME  observations,  instrumented  data  collection,  JTC 
output,  and  surveys  (i.e.,  prepared  questionnaires  and  interviews).  SME  observations  constituted 
the  bulk  of  the  data  collected  although  the  following  agencies,  all  of  which  collected  data  subject 
to  the  control  of  ATEC  (OTC),  contributed  key  data  elements.  BCTP  collected  observations  of 
the  EXFOR  across  each  of  the  battlefield  operating  systems  (BOS).  These  were  a  valuable 
source  of  corroborative  data  for  the  battle  analysis.  The  TRADOC  Program  Integration  Office  - 
ABCS  (TPIO-ABCS)  collected  operational  architecture  data.  The  TRADOC  Office  of  the 
Deputy  Chief  of  Staff  for  Training  -  West  (DCST-W)  generated  insights  about  training  support 
packages  (TSP).  The  Land  Information  Warfare  Activity  (LIWA)  collected  10  data  from  the 
World  Class  Opposing  Force  (WCOPFOR)  perspective  for  their  vulnerability  assessment  (VA). 
The  Command  and  General  Staff  College  (CGSC)  collected  data  on  emerging  tactics, 
techniques,  and  procedures.  The  Program  Executive  Office  for  Command,  Control  and 
Communication  Systems  (PE0-C3S),  using  MITRE,  collected  ABCS  performance  data.  IPs  and 
analysts  together  used  the  data  collected  during  the  DAWE  to  develop  issue-centric  insights. 

2.2  Execution  Phase 

The  following  paragraphs  describe  the  daily  rhythm  at  the  DAWE  including  the  data  collection 
process  and  the  analysis  from  both  the  warfighting  and  the  analytical  perspectives.  This  section 
also  describes  how  the  insights  from  these  two  perspectives  were  integrated. 

2.2.1  Data  Collection  Process 

ATEC  (OTC),  as  the  lead  agency  responsible  for  DAWE  data  collection,  developed  a  DAWE 
data  collection  plan  for  development  of  a  single  database  of  data  collected  from  the  DAWE. 
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This  plan  was  finalized  in  October  1997  after  extensive  meetings  and  workshops  with  issue 
proponents  and  analysts.  The  collection  plan  identified  the  data  to  be  collected,  collection 
methods  to  be  used  (observation,  instrumentation,  surveys,  etc.),  and  the  data  collection 
responsibilities  for  each  issue.  ATEC  (OTC)  trained  all  SMEs  prior  to  the  experiment  in  the 
FXXI  operational  concepts,  data  collection  and  observation  procedures  and  in  the  capabilities  of 
the  database  management  system  to  be  used.  During  the  DAWE,  the  SMEs  recorded  their 
observations  on  laptop  computers  that  they  were  issued  by  ATEC  (OTC).  Their  observations 
were  collected  (downloaded  to  the  computer  network)  every  six  hours  over  the  course  of  the 
DAWE.  ATEC  (OTC)  was  responsible  for  quality  oversight  of  the  data  collection  process, 
conducted  quality  control  checks  of  the  data  and  provided  the  data  to  analysts  and  IPs  for 
reasonableness  checks  prior  to  locking  the  data  into  the  database.  Other  data  collected  such  as 
that  collected  from  instrumentation,  surveys,  and  questionnaires  was  also  subjected  to  quality 
control  checks.  All  data  was  then  consolidated  into  an  automated  database  immediately  after  the 
DAWE  and  ATEC  (OTC)  provided  the  resulting  database  after  the  DAWE  to  the  analytic 
agencies  for  analysis.  ATEC  (OTC)’s  data  collection  process  was  modeled  after  their  data 
collection  process  typically  used  for  tests. 

2.2.2  Analysis  During  the  Experiment 

2.2.2. 1  Warfighting  Perspective 

Within  the  context  of  the  WFX  approach,  BCTP  conducted  a  traditional  battle  command 
evaluation.  They  utilized  OCs  to  assess  the  ability  of  the  EXFOR  to  manage  and  integrate  the 
BOSs  (maneuver,  fire  support,  intelligence,  air  defense,  command  and  control  (C2),  mobility  and 
survivability,  and  CSS).  The  TRAC  CAP  team  contributed  an  assessment  of  the  EXFOR’s 
execution  of  the  Force  XXI  patterns  of  operation  from  a  warfighting  perspective  to  the  BCTP 
evaluation.  The  BCTP  team  chiefs  worked  closely  with  the  CAP  team  to  share,  discuss,  and 
interpret  observations  and  derive  appropriate  insights.  Typically,  this  was  done  twice  daily, 
during  the  shift  change  meetings.  BCTP  presented  the  daily  warfighting  insights  at  the  nightly 
general  officer  update  briefings  and  at  the  after-action  reviews  (AARs)  during  the  AWE.  Upon 
completion  of  the  operational  phases  of  the  DAWE,  the  CAP  team  also  developed  insights  for 
presentation  at  the  general  officer  updates  in  terms  of  the  patterns  of  operation. 

2. 2. 2. 2  Analytical  Perspective 

The  analysis  effort  during  the  experiment  was  based  on  data  collected  from  various  sources.  The 
primary  data  collection  effort  used  over  100  SMEs  trained  and  managed  by  ATEC  (OTC)  and  60 
BCTP  OCs.  The  data  collected  by  ATEC  (OTC),  BCTP,  and  others  were  consolidated  into  an 
ATEC  (OTC)  database,  which  contained  over  6,000  SME/OC  observations,  interviews,  surveys, 
etc.  Daily  during  the  experiment,  analysts  from  TRAC,  ATEC’s  Army  Evaluation  Command 
(AEC),  and  the  Army  Research  Laboratory  (ARL),  with  assistance  from  the  issue  proponents, 
reviewed  the  observations  and  other  available  data  looking  for  trends  and  corroborating  data 
points  that  suggested  an  emerging  insight.  A  more  thorough  analysis  of  the  data  and  the  insights 
was  conducted  after  the  experiment  (as  discussed  in  paragraph  2.3).  The  Insights  Authentication 
Group  (IAG),  which  included  TRAC,  ARL,  and  ATEC  (AEC),  held  a  daily  meeting  to  screen  the 
emerging  insights  (developed  by  the  analysts  and  issue  proponents)  to  determine  viable, 
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supportable  analytical  insights.  The  IAG  reviewed  each  insight  based  upon  the  established 
criteria  of  significance  to  Army  leadership,  relevance  to  a  DAWE  issue,  completeness, 
consistency  with  other  observations,  and  whether  each  was  supported  by  observations.  Failing  to 
meet  these  criteria,  the  IAG  returned  the  insight  to  the  submitting  issue  proponent  and  analyst  for 
additional  research.  The  CAP  team  representative  at  the  daily  IAG  meetings  provided  the 
linkage  between  the  warfighting  and  analytic  teams  and  perspectives. 

2.2.3  Integration  of  Insights  at  the  Experiment 

The  CAP  team  was  the  primary  integrator  of  the  resulting  analytical  insights  with  BCTP’s 
insights  and  observations  to  determine  overall  insights  reflecting  both  the  warfighting  and 
analytic  perspectives  and  to  determine  potential  Force  XXI  capabilities.  The  analysts  and  issue 
proponents  conferred  on  a  continuing  basis  to  analyze  and  integrate  the  SME  observations.  If  the 
analyst  or  issue  proponent  found  any  problems  with  an  observation’s  completeness,  clarity, 
objectivity,  and  significance  they  submitted  an  anomaly  to  the  SME  asking  for  clarification, 
more  information,  or  additional  input  for  the  observation.  Once  any  problems  with  the 
observation  were  resolved,  the  lead  analyst  and  the  issue  proponents  reviewed  the  observations 
for  possible  integration  into  significant  results  called  insights.  After  the  IAG,  the  study  director 
and  the  senior  military  analysts  reviewed  the  analytical  and  CAP  team  insights,  selecting  those  to 
be  briefed  at  the  nightly  update  briefing. 

2.3  Post-Event  Phase 

The  first  product  of  the  DAWE  was  the  HR,  to  be  published  within  30  days  after  the  event.  To 
accomplish  this,  the  lead  analysts  developed  three  charts  immediately  after  the  event  and  before 
leaving  Fort  Hood  for  each  issue  category  to  identify  the  background,  key  emerging  insights 
based  on  available  data,  and  the  doctrine,  training,  leader  development,  organization,  materiel, 
and  soldiers  (DTLOMS)  implications  of  those  insights.  These  charts  were  consolidated  into  the 
HR.  Issue  proponents  and  analysts  then  returned  to  their  home  stations  and  continued  to  analyze 
the  data  and  to  refine  their  insights,  DTLOMS  implications,  and  battle  analysis.  This  process 
included  reviewing  all  of  the  data  to  confirm  the  emerging  insights  and  develop  any  new 
insights.  The  lead  analysts  then  determined  which  insights  were  key  and  appropriate  for 
inclusion  in  the  Final  Report.  This  data  included  the  SME/OC  observations  collected  at  the 
DAWE,  survey  results,  interviews,  instrumented  data,  JTC  output,  and  results  from  other 
organizations  gathering  information  (BCTP,  DCST-W,  TPIO-ABCS,  and  4ID).  A  DAWE  Final 
Report  workshop  was  held  at  Fort  Leavenworth  26-28  January  1998  at  which  issue  proponents, 
analysts,  and  other  organizations  gathering  information  at  the  DAWE  briefed  their  key  insights, 
answers  to  their  issues,  and  DTLOMS  implications  to  TRAC  and  other  participating  analytic 
organizations.  The  briefings  and  audience  discussions  facilitated  the  integration  of  the  results. 
The  issue  proponents  and  analysts  then  refined  their  input  based  on  the  discussions  at  the 
workshop  and  submitted  their  final  analytic  results  to  TRAC  for  inclusion  into  the  Final  Report. 

3.  Lessons  Learned 

This  section  of  the  paper  focuses  on  the  data  collection  and  analysis  processes  lessons  learned 
from  the  DAWE.  The  lessons  learned  are  based  on  a  data  collection  and  analysis  after  action 
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review  (AAR)  conducted  on  14  April  1998  at  Fort  Leavenworth,  Kansas.  Participants  in  the 
AAR,  and  those  providing  input,  were  TRAC  -  Fort  Leavenworth  (TRAC-FLVN),  ARL,  ATEC 
(AEC),  TRAC  -  Fort  Lee  (TRAC-LEE),  TRAC  -  White  Sands  Missile  Range  (TRAC -WS MR), 
and  ATEC  (OTC).  The  following  lessons  learned,  documented  from  the  perspective  of  the 
DAWE  Study  Director,  are  presented  for  each  of  the  three  phases:  planning,  execution,  and  post¬ 
event. 

3.1  Planning  Phase 

Data  collection  procedures  typically  used  for  operational  testing  should  not  be  strictly 
applied  to  AWEs.  Traditional  testing  protocols  were  applied  to  the  data  collection  procedures 
for  the  DAWE.  These  protocols,  in  some  cases,  impacted  the  subsequent  analysis  process. 
Three  particular  impacts  were  observed  as  a  result  of  this  approach.  First,  to  ensure  operations  in 
the  TOCs  were  not  hindered  with  an  excessive  number  of  visitors,  direct  observation  and  data 
collection  by  anyone  other  than  SMEs  formally  designated  and  trained  as  data  collectors  was 
restricted.  Direct  observations  by  analysts  and  issue  proponents  assisting  in  the  analysis  of 
issues  was  limited.  Second,  the  data  collected  during  the  DAWE  was  controlled  and  its 
distribution  restricted  (as  is  often  necessary  in  a  sensitive  test  of  a  developmental  system)  such 
that  the  timing  and  completeness  of  the  real-time  analysis  process  was  impacted.  The  translation 
of  subjective  observations,  once  obtained  from  the  SMEs,  was  difficult  for  the  analysts  who  had 
had  no  direct  observation  of  DAWE  events.  Third,  the  test-oriented  data  collection  process  was 
focused  at  too  low  a  level  of  detail  for  an  experiment  that  was  based  on  subjective  data  collection 
and  evaluation.  The  data  elements  to  be  collected  in  many  cases  were  too  detailed  and  too 
narrowly  focused  for  the  SMEs’  examination  of  the  division’s  command  and  staff  operations. 
This  resulted  in  a  large  number  of  data  elements,  leading  to  a  large  number  of  questions  for  many 
of  the  SMEs  to  answer.  Some  SMEs  had  up  to  300  questions  to  answer,  well  beyond  the  scope 
of  what  they  could  reasonably  do.  In  hindsight,  maintaining  the  collectable  elements  at  a  higher, 
more  functional  level  would  reduce  the  number  of  questions  that  future  SMEs  would  be  required 
to  answer. 

Senior  Army  leadership’s  guidance  is  needed  early  in  the  planning  phase  and  throughout 
the  process.  The  DAWE  planning  phase  began  while  the  preceding  TFXXI  AWE  was  also  still 
in  its  planning  process.  The  attention  of  the  senior  leaders  was  clearly  focused  on  that  earlier 
significant  event.  Attention  turned  to  the  DAWE  only  after  the  TFXXI  AWE  was  completed. 
The  DAWE  suffered  because  of  this  lack  of  attention.  In  particular,  the  issue  development 
process  lacked  senior  leaders’  direction.  Senior  leaders’  guidance  was  needed  to  narrow  the 
focus  of  each  of  the  eight  overarching  issues,  which  would  have  facilitated  the  issue 
development  process.  The  process  of  obtaining  senior  leaders’  guidance,  through  the  process 
action  team  (PAT)  meetings  conducted  by  the  JVO  of  Headquarters  TRADOC,  was  inadequate. 
At  the  PAT  meetings,  each  in-process  team  (IPT)  lead  (e.g.,  analytic  lead,  scenario  lead,  etc.) 
presented  issues  and  concerns  that  the  JVO  was  to  elevate  to  the  EWG  for  guidance  or  approval. 
Often  times,  guidance  was  not  obtained  and/or  not  conveyed  back  to  the  IPT  lead.  Increased 
involvement  of  senior  leaders,  particularly  in  the  planning  phase,  would  have  resulted  in  an  even 
better  experiment.  Also,  senior  leaders’  guidance  is  required  to  help  enforce  the  good  idea  cut¬ 
off  date  (GICOD)  and  everything  in  place  date  (EIPD).  For  instance,  PEO-C3S  was  allowed  to 
modify  ABCS  even  after  SIMEX  I,  which  was  long  after  the  GICOD  and  EIPD.  These  changes 
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in  ABCS  caused  NSC  to  make  last  minute  changes  in  the  interface  modules  between  the  JTC  and 
ABCS  and  also  hindered  the  training  of  soldiers  on  ABCS,  which  affected  the  performance  of 
the  unit. 

Establish  organizational  responsibilities  early  in  the  process.  All  participants  in  an  AWE 
need  to  know  their  specific  role  and  the  role  of  others  to  work  effectively.  For  example,  in  the 
DAWE,  the  role  of  BCTP  and  their  relationship  with  TRAC’s  CAP  team  conducting  the  battle 
analysis  was  not  established  until  very  late  in  the  process  (SIMEX  II)  although  this  relationship 
quickly  developed  and  proved  to  be  one  of  the  most  positive  accomplishments  of  the  analytical 
effort.  Additionally,  the  responsibilities  of  some  agencies  (e.g.,  PEO-C3S,  CGSC)  collecting 
data  from  the  DAWE,  and  their  relationship  with  the  lead  for  data  collection,  ATEC  (OTC),  was 
never  well  established.  This  lack  of  defined  relationships  caused  problems  in  data  collection  and 
analysis.  These  problems  included  some  agencies  not  providing  their  observations  from  the 
DAWE  for  inclusion  in  the  official  database,  agencies  not  providing  information  in  accordance 
with  established  formats,  and  a  general  lack  of  recognition  of  ATEC  (OTC)’s  preeminent  role  in 
data  collection. 

Establish  consistent  relationship  among  analysts,  testers,  evaluators,  and  issue  proponents. 

TRAC,  as  the  lead  agency,  directed  that  the  analysis  teams  rely  upon  the  issue  proponents  for 
functional  expertise  in  the  analysis  process.  TRAC  would  then  be  responsible  for  the  analytic 
validity  and  disconnecting  any  proponent  parochialism.  However,  for  some  issue  areas,  the 
relationship  with  issue  proponents  was  not  sufficiently  established  to  allow  the  issue  proponents 
to  participate  in  the  analytical  process.  As  a  result,  several  issue  proponents  disagreed  with  the 
contents  of  the  feeder  report  forcing  the  study  director  to  resolve  these  conflicts  after  the  fact 
rather  than  as  a  part  of  the  analysis  process.  This  problem  can  be  avoided  in  future  AWEs  by 
establishing  consistent  relationships  between  analytic  agencies  and  issue  proponents  which  lead 
to  closer  cooperation  among  the  participating  agencies. 

Firmly  establish  the  authority  of  the  lead  analytic  agency  and  its  study  director.  Although 
TRAC’s  authority  as  the  DAWE  lead  analytic  agency  became  well  known  to  the  participating 
data  collection  and  analysis  agencies,  some  DAWE  participants  did  not  initially  understand 
TRAC’s  responsibility.  Some  agencies,  particularly  those  resident  at  Fort  Hood,  habitually  work 
with  data  collection  and  analysis  agencies  other  than  TRAC.  Initial  lack  of  understanding  of 
TRAC’s  role  in  the  DAWE  resulted  in  TRAC  occasionally  experiencing  delays  in  obtaining 
timely  information  regarding  events  impacting  on  the  DAWE  analysis.  In  most  instances  these 
problems  were  resolved  through  further  coordination  among  the  data  collection  and  analysis 
agencies,  but  in  some  cases  they  were  not.  Firm  and  early  establishment  of  the  lead  analytic 
responsibilities,  that  is,  who  is  in  charge  overall,  is  required  to  circumvent  these  problems. 

Define  a  common  lexicon  of  terms  across  agencies.  Throughout  the  DAWE  planning  and 
execution,  there  was  confusion  regarding  the  different  use  of  the  terms;  issue,  essential  elements 
of  analysis  (EEA),  measures  of  effectiveness  (MOE)  and  performance  (MOP),  and  collectable 
data  elements.  It  took  considerable  time  to  realize  that  there  was  a  problem  and  to  come  to  some 
agreement  on  the  use  of  this  terminology.  More  importantly,  it  became  apparent  that  these 
inconsistencies  were  affecting  the  ability  to  properly  define  the  scope  of  the  experiment.  For 
example,  the  analysis  community  typically  defines  the  MOE  and  MOP  as  the  lowest  elements  to 
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be  collected  to  assist  the  analyst  in  answering  the  sub-issues  and  EEA.  However,  in  the  testing 
environment,  MOPs  are  used  to  further  define  the  MOEs.  Data  elements,  used  to  further  define 
the  MOPs,  are  the  lowest  element  collected  in  the  testing  environment  (e.g.,  number  of  times  the 
system  is  used  per  hour).  The  confusion  over  these  terms  resulted  in  the  analytic  teams  adjusting 
and  readjusting  their  data  collection  requirements.  A  common  lexicon  of  terms,  appropriate  for 
the  experiment,  should  be  established  and  agreed  to  early  in  the  experimentation  planning 
process. 

Identify  Warfighter  Rapid  Acquisition  Program  (WRAP)  candidates  and  initiative  data 
collection  requirements  early  in  analysis  planning.  WRAP  candidates  to  be  played  in  the 
DAWE  were  not  identified  until  immediately  preceding  the  DAWE.  Additionally,  some  of  the 
initiatives  played  in  the  DAWE  were  not  available  to  the  EXFOR  in  time  for  the  conduct  of 
proper  training.  This  made  the  data  collection  and  evaluation  of  these  initiatives  and  WRAP 
candidates  difficult  and,  in  some  cases,  impossible.  Early  identification  of  data  collection 
requirements  for  these  systems  is  required.  Once  the  WRAP  candidates  are  fielded  and  the 
soldiers  are  trained,  modifications  to  the  systems  must  be  kept  to  a  minimum  and,  if  made, 
followed  by  the  necessary  training.  Early  availability  of  the  WRAP  candidates  to  the  EXFOR 
and  identification  of  data  collection  requirements  for  the  systems  is  required. 

Simulation  exercises  (SIMEXs)  preceding  the  DAWE  were  invaluable  in  supporting 
planning  for  the  data  collection  and  analysis  process,  but  more  time  between  them  and  the 
AWE  is  required.  SIMEX  I,  conducted  in  June  1997,  and  SIMEX  II,  conducted  in  September 
1997,  provided  excellent  opportunities  to  field  trial  data  collection  and  analysis  procedures.  This 
included  determining  the  detail  and  focus  of  the  data  and  how  it  should  be  collected.  However, 
sufficient  time  did  not  exist  after  these  events  to  react  fully  to  lessons  learned  from  them  prior  to 
the  next  event.  For  example,  there  were  problems  identified  during  SIMEX  I  in  the  observations 
database  software  and  the  insights  database  software  (lack  of  flexible  database  search 
capabilities)  to  be  used  to  support  the  data  collection  and  analysis  effort.  There  was  not  enough 
time,  however,  before  the  DAWE  to  finish  development  and  verification  of  some  of  the  proposed 
changes  that  would  have  improved  the  data  collection  and  analysis  effort.  As  the  Army  conducts 
other  AWEs,  however,  data  collection  and  analysis  procedures  will  improve  as  these  supporting 
systems  mature. 

SMEs  should  be  selected  who  have  no  vested  interest  in  experimental  outcomes.  Generally, 
SMEs  shared  the  interests  of  the  TRADOC  school  or  center  they  represented.  Generally,  the 
SMEs  were  located  at  a  particular  location  within  the  4ID  and  were  required  to  answer  many 
issues.  Although  issues  spanned  several  proponent  areas,  most  SMEs  were  clearly  focused  on 
their  own  TRADOC  school  or  center’s  issues.  It  became  obvious  that  it  was  very  hard  for  a 
SME  to  completely  remove  branch  influence  from  their  observations.  A  potential  alternative 
source  of  SMEs  for  AWEs  would  be  non-participating  Forces  Command  (FORSCOM)  unit(s), 
reserve  components,  retired  members,  and  civilians.  The  SMEs  should  serve  only  as  consultants 
to  the  analytic  team. 

Analysts  must  be  more  involved  in  the  development  of  surveys,  questionnaires  and 
interviews.  The  surveys,  questionnaires  and  interviews  were  developed  by  ATEC  (OTC),  with 
input  from  the  issue  proponents,  to  address  questions  from  each  study  issue  category.  The 
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surveys,  questionnaires  and/or  interviews  were  consolidated  across  each  study  issue  category 
such  that  only  one  integrated  instrument  was  developed  for  each  soldier  respondent.  For 
example,  similar  questions  posed  from  different  issue  proponents  regarding  different  issue 
categories  were  consolidated  into  one  question  for  the  integrated  instrument.  However,  even 
though  the  questions  were  sometimes  similar  across  categories,  the  perspective  from  various 
issue  proponents  was  different.  Consolidating  these  questions  lost  the  nuances  of  the  issue 
proponents  that  developed  them.  Additionally,  the  EXFOR  set  a  limit  on  the  number  of 
questions  that  could  be  administered  to  EXFOR  soldiers.  The  process  of  developing  these 
instruments  is  more  complicated  than  simply  including  all  questions  and  then  reducing  the 
number  to  an  arbitrary  total.  It  was  difficult  to  determine  the  most  important  questions  from 
each  of  the  categories,  thereby  issue  proponents  were  forced  to  reduce  the  number  of  their 
questions  to  a  predetermined  limit  without  regard  to  the  relative  importance  of  the  category.  The 
effective  development  and  use  of  these  instruments  requires  more  coordination  with  the  analysts 
to  help  reduce  duplication  and  still  ensure  that  questions  will  elicit  the  desired  information. 
Additionally,  early  coordination  with  the  EXFOR  would  help  alleviate  arbitrary  limitations  in 
questioning  its  soldiers. 

The  experimentation  can  benefit  from  pre-  and  post-exercise  modeling,  if  sufficient  time  is 
available.  The  DAWE,  like  any  other  experiment  that  is  conducted  as  part  of  a  warfighter 
exercise,  was  constrained  analytically  because  the  exercise  was  not  repeatable  in  a  live 
environment  due  to  time  limitations  and  cost  constraints.  This  reduced  the  potential  to  examine 
specific  issues  that  require  some  comparison  of  alternatives,  to  include  any  comparison  to  a 
baseline  capability.  Typically,  this  limitation  is  overcome  through  employment  of  a  model- 
exercise-model  (M-E-M)  approach  where  pre-exercise  modeling  is  conducted  to  help  focus  the 
data  collection  and  where  alternatives  are  examined  in  a  constructive  model  after  the  exercise. 
This  approach,  for  example,  was  done  as  part  of  the  TFXXI  AWE.  The  DAWE,  however,  was 
followed  almost  immediately  by  a  Board  of  Director  (BOD)  meeting  where  decisions  regarding 
Force  XXI  would  be  made  based  on  DAWE  results.  This  prohibited  consideration  of  employing 
a  M-E-M  approach  which  would  have  helped  focus  the  data  collection  and  would  have  enabled 
comparison  of  the  EXFOR  to  a  baseline  Army  of  Excellence  (AOE)  division,  as  well  as 
examination  of  additional  issues.  Future  experiments,  however,  should  consider  use  of  the  M-E- 
M  approach,  unless  time  constraints  prohibit  it. 

3.2  Execution  Phase 

Employ  a  data  collection  and  analysis  process  that  provides  the  flexibility  to  respond  to 
unforeseen  scenario  events  and  will  enable  the  capturing  of  emerging  insights  from  a 
warfighting  perspective.  Given  that  one  can’t  plan  for  every  eventuality,  a  process  was  put  in 
place  to  collect  the  data  from  warfighting  perspective  and  react  to  unplanned  scenario 
opportunities  for  collecting  insights.  The  CAP  team  was  used  to  accomplish  this  in  the  DAWE. 
Prior  to  the  DAWE  experiment,  the  CAP  team  identified  certain  scenario  events  that  could 
potentially  impact  key  decisions  in  the  battle.  By  knowing  where  and  when  to  look,  members  of 
the  CAP  team  were  able  to  position  themselves  in  various  cells  to  observe  operations  during 
these  scenario  events  and  identify  any  insights.  Additionally,  the  CAP  team  was  called  upon  to 
collect  information  and  analyze  events  unfolding  in  the  free-play  scenario  that  were  not  foreseen 
prior  to  the  experiment.  For  example,  when  chemical  weapons  were  used  by  the  WCOPFOR, 
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this  team  was  dispatched  to  observe  the  results.  The  data  collection  and  analysis  process  must 
incorporate  an  evaluation  from  a  warfighting  perspective  and  have  flexible  procedures  in  place  to 
handle  these  unforeseen  experimental  opportunities,  to  include  the  flexibility  to  rapidly  change 
and  tailor  data  collection  and  analysis  approaches  during  an  AWE. 

Subjective  observations  from  an  AWE  do  not  require  the  same  level  of  control  as  data  from 
a  formal  operational  test.  During  the  DAWE,  all  observations  collected  by  the  SMEs  were 
subjected  to  multiple  validity  checks  by  not  only  ATEC  (OTC),  but  by  analysts  and  issue 
proponents  as  well.  Some  of  these  repetitive  checks  were  unnecessary  and  slowed  the  data 
collection  process.  This  was  recognized  during  SIMEX  I  and  adjustments  were  made  to  the  data 
collection  process  to  reduce  the  number  of  validity  checks  on  the  data,  but  this  was  not  sufficient. 
While  timeliness  or  any  other  measure  of  quality  for  subjective  data  should  not  be  forfeited,  the 
same  quality  control  standards  used  for  formal  system  tests  are  not  necessary.  Also,  the  closely 
held  ownership  of  the  data  by  data  collectors  limited  the  analysts’  access  and  ability  to  review 
and  analyze  the  data  during  the  DAWE.  The  extent  to  which  subjective  data  is  controlled  in 
future  AWEs  should  be  re-evaluated.  The  control  of  the  data  should  not  limit  the  analytic 
process. 

There  should  be  a  single,  consolidated  data  collection  plan  to  produce  a  single  database  for 

use  by  all.  Although  this  was  always  a  stated  objective  for  the  DAWE,  some  organizations  still 
conducted  their  own  data  collection  effort  to  meet  their  own  purposes.  Consequently,  the  data 
collected  were  not  included  in  the  one  formal  DAWE  database  produced  by  ATEC  (OTC)  for 
use  by  the  analysts  and  the  issue  proponents.  This  occurred  despite  several  attempts  by  ATEC 
(OTC)  and  TRAC  to  formally  integrate  the  data  collection  activities  of  all  concerned. 

The  formatting  of  the  data  input  into  the  DAWE  database  must  be  controlled.  The  DAWE 
database,  because  of  its  vast  size,  had  a  very  slow  load  and  response  time.  The  database,  for 
example,  had  some  inordinately  large  files  (e.g.  several  animated  PowerPoint  slide  files  of  150 
megabytes  each).  Additionally,  several  fragmentary  orders  (FRAGOs)  were  input  into  the 
database  as  bit-mapped  pictures.  This  took  up  a  lot  of  space  and  affected  the  response  speed  of 
the  database  manager.  A  1000- fold  reduction  in  file  size,  and  a  corresponding  reduction  in 
access  time,  would  reasonably  be  possible  if  they  had  been  input  as  text  files. 

Analysts  should  participate  in  the  interview  process.  Immediately  after  the  DAWE,  the 
Division  Commander,  Assistant  Division  Commanders,  the  Chief  of  Staff,  primary  staff,  and 
selected  0-6  level  commanders  were  interviewed  as  part  of  the  formal  data  collection  process. 
These  interviews  were  conducted  with  restricted  participation  from  the  issue  proponents  and 
analysts,  again  a  carryover  of  formal  test  protocols.  Because  of  this  restricted  process,  analysts 
were  not  allowed  to  follow  up  on  the  responses  of  the  interviewees.  Analysts’  participation  in 
the  interview  process  would  enable  them  to  ask  spontaneous  questions  in  response  to 
interviewees’  answers  and,  thus,  glean  more  information. 

The  Center  for  Army  Lessons  Learned  Collection  and  Observation  Management  System 
(CALLCOMS)  database  management  system  (DBMS)  did  not  efficiently  handle  the  large 
AWE  database  of  observations.  Analysts  discovered  in  SIMEX  I  that  CALLCOMS  was  not 
able  to  efficiently  handle  the  number  of  SME  observations  (up  to  3000  daily)  expected  from  the 
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DAWE.  CALLCOMS  took  an  average  of  90  seconds  to  display  a  single  observation  once  the 
display  sequence  was  initiated.  Hence,  each  analyst  would  not  have  time  to  view  the  number  of 
observations  expected  each  day.  ATEC  (OTC)  developed  a  replacement  DBMS  called 
“TexComs”  which  was  a  great  improvement  in  responsiveness  over  CALLCOMS.  TexComs 
was  able  to  handle  over  6000  observations  from  the  DAWE  with  little  reduction  in  speed  of 
transactions.  TexComs  adequately  captured  SMEs  observations  and  was  user  friendly  with 
searches  and  features  that  met  most  analytical  needs.  Any  replacement  DBMS  should  have  at 
least  the  features  of  TexComs.  However,  TexComs  lacked  the  flexibility  to  do  complex  searches 
combining  and/or  excluding  specific  fields  that  should  be  added  to  any  DBMS  used  in  future 
AWE  efforts. 

Analysts  should  be  permitted  to  serve  as  SME  observers.  Analysts  could  fill  the  positions  of 
both  SME  observer  and  analyst  by  being  granted  unlimited  access  to  the  various  locations  within 
their  area  of  responsibility.  Access  by  analysts  to  various  EXFOR  headquarters  was  restricted 
because  of  the  need  to  limit  the  number  of  observers,  or  overall  number  of  personnel,  in  the 
EXFOR’ s  TOCs.  Additionally,  application  of  formal  test  protocols  did  not  allow  any  analyst  to 
directly  interface  with  a  SME  when  there  was  a  problem  (i.e.,  an  anomaly)  with  an  observation. 
This  lack  of  direct  interface  forced  the  analysts  to  write  instructions  and  inquiries  back  to  the 
SMEs  to  resolve  the  anomaly.  The  anomaly  resolution  process  took  one  to  three  days  for  an 
answer  to  come  back  to  the  analysts.  In  some  cases,  it  took  more  than  one  request  to  get  the 
information  desired.  This  cumbersome  process,  which  caused  analysis  results  to  be  delayed  in 
some  cases  well  beyond  when  significant  events  occurred  in  the  scenario,  could  have  been 
avoided  if  analysts  could  have  also  served  as,  or  along  side,  SME  observers.  If  planned  for  early 
enough,  this  also  would  help  to  lower  the  number  of  SMEs  needed,  which  was  a  concern  and 
expense  for  the  data  collection  community. 

SMEs  should  be  focused  on  issues,  not  on  a  geographical  location.  SMEs  were  severely 
restricted  because  of  space  limitations  on  where  they  could  go  in  the  TOCs.  Because  of  the  flow 
of  information  and  coordination  required  among  the  TOCs,  the  issues  of  staff  effectiveness,  10, 
and  C2  cannot  be  resolved  by  observing  only  one  TOC.  Scheduled  meetings  among  the  SMEs 
were  also  not  sufficient  to  permit  the  necessary  exchange  of  information.  Most  SMEs  need  to  be 
able  to  change  locations  to  follow  through  on  observations.  SMEs  could  have  been  more 
effective  if  they  were  organized  much  like  the  CAP  team,  or  perhaps  like  the  BCTP  OCs,  under 
the  leadership  of  the  issue  category  analyst,  and  targeted  to  go  where  they  needed  to  go. 

The  insight  development  process  should  be  scenario  event  based  rather  than  timeline 
based,  and  addressed  by  a  team  of  analysts  similar  to  the  CAP  team  employed  in  the 
DAWE.  Analysts  experienced  frustration  in  trying  to  develop  insights  on  a  periodic  basis  to 
support  a  daily  briefing,  the  timing  of  which  had  little  to  do  with  what  was  happening  in  the 
scenario.  Insights  were  based  on  scenario  events,  so  the  reporting  should  also  be  on  a  scenario 
event  basis.  Periodic  (daily)  reporting  may  give  significance  to  an  item  when  there  may  not  be 
anything  significant  yet  to  report.  Additionally,  the  development  of  insights  should  be  done  by  a 
team  of  analysts  focused  on  scenario  events,  such  as  the  CAP  team  employed  in  the  DAWE, 
rather  than  strictly  by  analysts  looking  at  specific  issues,  or  a  category  of  issues. 
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There  must  be  a  more  complete  understanding  of  the  analytical  objectives  of  an  AWE. 

Although  the  exception  rather  than  the  rule,  the  focus  of  some  in  the  EXFOR  was  strictly  on 
training  (the  fighting  and  winning  of  the  battle),  rather  than  on  the  answering  the  issues  of  the 
experiment.  For  example,  the  Engineer  Brigade  Commander,  while  not  part  of  the  Force  XXI 
division  Table  of  Organization  and  Equipment  (TOE),  continued  to  participate  in  the  DAWE  at 
the  request  of  the  Division  Commander  even  though  this  limited  the  ability  of  the  analysis  team 
to  address  the  issues  associated  with  the  loss  of  the  engineer  brigade  headquarters.  The  data 
collected  on  engineer  C2  within  the  division  TOCs  were  skewed  by  the  presence  of  an  engineer 
brigade  commander.  Therefore,  no  conclusive  evidence  was  collected  on  the  adequacy  of 
engineer  C2  and  staffing  within  the  TOC.  Additionally,  the  EXFOR  controlled  the  composition 
of  the  questionnaires,  surveys,  and  interviews  and  limited  their  application  to  selected 
individuals.  The  limitation  placed  on  this  collection  effort  limited  the  data  available  to  the 
analyst  and  proponents.  These  instances  of  training  emphasis  tended  to  overshadow  the 
analytical  requirements  of  the  AWE  and  presented  challenges  to  the  study  team.  A  more 
complete  understanding  of  analytical  objectives  by  the  entire  EXFOR  would  enable  an  AWE  to 
better  meet  the  primary  objectives  of  both  the  training  and  analytic  communities. 

3.3  Post-Event  Phase 

There  must  be  allowances  for  analysis  to  be  done  concurrently  with  the  AWE.  In  order  to 
be  responsive  to  its  reporting  requirements,  the  DAWE  analysis  team  did  much  of  its  analysis 
concurrently  with  the  DAWE  execution.  This  worked  well  despite  some  natural  discomfort 
among  selected  analysts  regarding  the  need,  in  some  cases,  to  analyze  incomplete  data  before  the 
end  of  the  DAWE.  It  also  enabled  the  study  team  to  develop  the  daily  insights  briefing  and  to 
produce  the  HR  within  thirty  days  after  the  DAWE. 

The  study  director  should  use  a  post-AWE  workshop  to  develop  the  final  report.  The 

DAWE  Final  Report  was  developed  by  integrating  the  written  input  from  the  participating 
analytical  agencies  after  they  had  returned  to  their  home  stations  from  the  DAWE.  Any 
revisions,  conflicts,  or  questions  had  to  be  resolved  via  FAX,  telephone,  and  email,  which  caused 
delays  in  answering  and  coordinating  refinements  to  the  report.  Furthermore,  certain  analysis 
team  members  being  transferred,  retired,  or  assigned  other  projects  exacerbated  these  delays. 
The  study  director  should  hold  a  workshop  with  all  analytic  agencies  immediately  after  the 
analysis  is  complete  to  develop  the  final  report.  This  workshop  would  facilitate  the  cross¬ 
fertilization  of  ideas  and  provide  an  opportunity  for  resolving  contradictory  insights. 

The  HR  should  be  recognized  as  an  emerging  impressions  report,  not  a  comprehensive 
analytical  report  for  an  AWE.  The  DAWE  HR  was  so  comprehensive,  with  over  75  pages, 
that  once  it  was  published,  it  ran  the  risk  of  being  perceived  by  many  as  the  final  report  based  on 
a  completed  analysis.  The  quick  response  of  thirty  days  to  complete  the  HR  required  analysts 
and  issue  proponents  to  conduct  analysis  of  incomplete  and  emerging  data  to  develop  the  initial 
input  to  the  HR.  While  there  were  no  insights  in  the  HR  that  were  contradicted  after  completion 
of  the  analysis,  this  possibility  existed.  If  immediate  feedback  is  required  in  addition  to  a 
comprehensive  final  report,  then  a  short  memorandum  of  emerging  impressions  should  be 
adequate.  Producing  only  one  comprehensive  report  would  allow  more  time  for  the  analysis 
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team  to  review  the  completed  database,  resolve  any  conflicts,  determine  the  key  insights,  and 
staff  the  final  report. 

An  AWE  database  should  be  structured  to  best  facilitate  the  retrieval  of  all  data.  The 

DAWE  database  structure  did  not  account  for  the  conditions  in  the  DAWE  that  were  different 
from  the  normal  testing  environment.  These  conditions  included  time  constraints,  multiple 
sources,  and  the  diverse  character  of  the  data.  This  resulted  in  a  format  for  the  DAWE  database 
that  included  a  multitude  of  text  files,  smaller  databases,  spreadsheets,  and  graphics  files. 
Because  of  this,  the  dendritic  of  directory  names,  sub-directory  names,  and  file  names  were  not 
always  descriptive  of  the  data  they  contained.  This  posed  a  problem  to  the  analysts  in  finding  the 
data  and  understanding  its  source  and  its  relationship  to  the  study  issues.  The  database  for  an 
AWE  should  be  more  flexible  to  accommodate  these  conditions  and  provide  a  more  intuitive, 
easily  understood,  and  more  accessible  overall  database. 

Tools  should  be  developed  as  part  of  the  Joint  Training  Confederation  (JTC)  and  used  in 
the  AWE  to  easily  generate  basic  effectiveness  information  from  the  exercise. 

The  JTC,  as  a  collection  of  training  simulations  that  use  unclassified  and  often  aggregated  data, 
does  not  generate  effectiveness  data  that  is  useful  to  the  analysts.  It  does,  however,  produce 
information  of  general  importance  which  can  be  of  use  to  the  analysts  (e.g.,  recording  the  history 
of  the  battle,  tracking  movement  of  units  and  the  forward  line  of  own  troops  (FLOT),  and 
recording  of  basic  unit  maneuvers).  In  the  DAWE,  Vision  21  (an  instrumented  JTC  after  action 
review  (AAR)  tool)  provided  this  information  as  well  as  damage  reports,  unit  strength  over  time, 
mission  and  posture  information,  and  equipment  lists  for  each  unit.  Vision  21,  or  another  similar 
tool,  should  be  further  enhanced  and  used  in  the  future  to  assist  the  analytical  community.  The 
enhancements  should  include,  as  a  minimum,  providing  information  on  the  units  in  “red  zone” 
fights  (e.g.,  which  units  engaged,  when,  where,  how  long,  what  systems  and  ammunition  used, 
etc.),  providing  the  ability  to  graphically  examine  the  battlespace  down  to  a  resolution  of  10 
kilometers  by  10  kilometers,  and  providing  equipment  lists  down  to  the  smallest  size  unit 
portrayed  in  the  training  model. 

4.  Summary 

In  every  successful  operation,  there  are  aspects  that  could  have  been  better  executed  to  achieve 
the  objectives.  This  is  especially  true  for  new,  evolving  ventures,  such  as  Army  advanced 
warfighting  experiments.  The  planning,  execution,  and  reporting  of  data  collection  and  analysis 
that  underpinned  the  DAWE  fulfilled  the  experimental  objectives,  but  still  should  be  refined  and 
changed  to  improve  them  for  future  AWE.  This  paper’s  lessons  learned  provide  the  basis  for  the 
necessary  changes.  The  lessons  learned  strongly  reinforce  the  need  for  the  AWE  planners,  data 
collectors,  and  analysts  to  be  innovative,  agile,  and  adaptive  throughout  the  planning  and 
execution  of  the  AWE  process.  The  following  paragraphs  summarize  the  key  lessons  learned. 

The  data  collection  procedures  routinely  used  for  live  operational  tests  in  the  field  are  not 
directly  applicable  to  AWE.  Limited  adaptation  of  established  procedures  was  made  for  the 
DAWE  and  further  adaptation  is  required  for  future  AWE. 
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Data  collection  should  be  a  unified  and  synchronized  effort  under  the  direction  of  a  single 
agency.  Quasi-independent  data  collection  by  multiple  agencies  took  place  during  the  DAWE 
and  should  not  be  permitted  in  future  AWE. 

Careful  and  deliberate  consideration  should  be  given  to  the  selection  of  the  database 
management  system  with  particular  attention  to  the  unique  data  processing  and  analysis  needs  of 
the  AWE.  Rather  than  simply  a  data  repository,  it  must  readily  enable  and  enhance  the 
attainment  of  the  analysis  objectives. 

Analysts  must  have  unrestricted  contact  and  interface  with  the  data  collectors  and 
observers/controllers  who  are  working  “in-the-box”  (i.e.,  simulated  environment)  during  the 
AWE,  to  include  entering  “the  box”  themselves  as  well  as  serving  in  a  data  collection  capacity  if 
necessary. 

Because  the  typical  AWE  is  a  one-time  event  and  the  scenario  is  allowed  to  develop 
dynamically,  the  data  collection  effort  must  be  able  to  rapidly  respond  to  targeted  areas  of 
interest  that  are  identified  and  cued  as  events  unfold.  Therefore,  procedures  should  be  developed 
to  enable  two  types  of  data  collectors  for  an  AWE:  those  who  are  fixed  in  place  (i.e.,  single 
physical  location)  to  observe  events  in  their  assigned  subject  area,  and  those  who  are  able  to 
track  subsequent  events  (i.e.,  physically  move)  triggered  by  and  related  to  an  initial  event 
occurring  in  their  subject  area. 

The  final  analysis  and  report  must  be  concluded  quickly  following  the  AWE,  while  first-hand 
information  is  readily  accessible  from  the  many  participants  prior  to  their  departure  or 
reassignment  to  other  duties. 
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